In Part 1 of this post, we looked at some properties that agents should have if they want to benefit consumers, and what communication between those agents and companies might look like. In this post, we’ll propose a custom architecture for an AI agent that specializes in helping consumers with customer experience journeys.
Building Blocks of Effective Agentic Systems
Simply pointing a LLM at a company and prompting it to advocate for the user is not likely to produce robust and reliable results. These are a proposed set of system components aimed at constructing a reliable, transparent, and respectful agentic system.
1) Ledger: A mutable set of goals that define what the agent is trying to accomplish. This would be generated at the beginning of the request and modified if the agent accomplishes something or gets blocked.
- Advanced models like OpenAI’s o1 use a private chain-of-thought to plan and break down tasks, but these are not persisted. We need agents to start with a plan and follow through over many interactions that could take weeks or months to complete.
2) Interaction Log: A reliable and immutable record of interactions to ensure accountability and traceability. The interaction log in this system would record actions the principal has taken (e.g., granting permission for the agent to use PII, declining consent for a course of action) and actions the agent has taken.
- This includes interpretations made by the agent, such as understanding language from the principal or counterparty.
- This log would be used in disputes to show when and how the user consented to decisions made by the agent, when those situations come up.
3) Supervisor/Subordinate/Scribe Agent Subnetwork: Implementing generative-adversarial relationships between agents to balance creativity and oversight. A single agent responsible for both safety and achieving favorable outcomes will not perform as well as separate agents that can coordinate and check each other.
- The safety agent can check and block the active agent from doing things it shouldn’t. Users can adjust and inform the safety agent’s mandate.
- A third scribe agent will examine the dialogue between the other two and note the final decision and reasoning in the interaction log.
- The supervisor and subordinate agents should always output “reason for doing something + decision.”, where “decision” is related to a verb from the protocol described in Part 1. While advanced models like o1 may not need this structured output, architecting the system this way allows for scaling down model costs using simpler models like GPT-4-mini.
4) Permissions Granting and Consent Gathering: A secure mechanism to manage and protect Personally Identifiable Information (PII) and other sensitive data.
- The system should restrict what PII it shares with LLM-based agents, and users should be able to retract that PII at any time.
- Users will need to allow agents access to data that companies possess. If a company implements OAuth or GNAP, we can mint tokens with scopes corresponding to the information they allow to be shared.
- Outside of these structured protocols, SMS/email verification can be used to prove user consent for data sharing.
5) State Machine: A structured framework to govern the states and transitions of the system, so the agent can activate itself in response to events or time passing.
- This is especially important for managing asynchronous processes with long gaps between events, such as emails or company-side request processing.
- We can use the state machine to track where the process is, and based on that state prompt users or companies to take action. A structured notion of state is also useful for expiring requests that are stuck after too long, or retrying unresolved actions.
- Structured states give us an easy way to show the user what’s going on with their request at a glance that means the same thing across all requests.
6) Tools: These agents need to interact with the world, whether that interaction is over email, phone calls, or a structured machine-to-machine protocol. We have a lot of experience with this from building Permission Slip in both automated and human-powered approaches to scaling data access requests.
Looking Ahead
The system considerations and components described here are just a starting point for discussion and experimentation. Human power is often more leveraged than automation, so we may be building more low-tech systems and incorporating more agentic architecture over time.
Some open questions we’ll be looking at next include:
-
- How, at a technical level, will machine agents communicate with each other once we are operating consensually with companies?
- What are the details of the high-level protocol that agents will use to interact?
- How exactly do we perform authorization in non-consentual company interactions?
- How do we best spend our innovation tokens to get agents like this working for consumers ASAP?
As we explore this space, we want to hear from you! If you’re a potential user or represent a company looking to get involved, drop us a line at innovationlab@cr.consumer.org with any questions or feedback.
